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NAEP: A NATIONAL DATA SYSTEM FOR THE 2 1ST CENTURY 

C. Phi 1 i p Kearney 
The University of Michigan 



liitroduct i on 

Information about tne condition of education — about the quality 
of schools, school districts, and state educational systems— has 
become a priority concern for the increasing array of new policy 
actors now involved in making decisions that affect the quality of the 
education being received by the Nation's children and youth. 
Unfortunately, however, current data systems are not capable of fully 
meeting these rapidly expanding needs for information about education. 
At best, they represent an unar t i cul ate?J set of discrete projects 
rather than a program or system for providing comprehensive, 
integrated, representative, accurate, and timely information on the 
elementary and secondary schools of the Nation (Silverman and Taueb^r, 
1985; Hall, Jaeger, Kearney, and Wiley 1985). 

However, in our view, one of these discrete pojects— the National 
Assessment of Educational Progress <NAEP)~more than any other 
existing data project has Inherent within it the potential to become 
a truly national data system and to meet the rapidly expanding needs 
for educational Information. The present moment, when the future 
configuration of NAEP is belrvg debated and decided, offers an 
excellent opportunity to move to develop NAEP Into such a data 
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system-a systerr. that will provide comprehensive, integratecS 
represenUtlve, accurate, and timely information on elementary and 
secondary education in these Un i ted States , and a system that .ill be 
up to meeting the information challenges of the 21st Century. 

NAEP, in its original configuration, did not have such potential 
due, in large part, to the political constrainf:s of the time. 
Messick, Beaton, and Lord <1983) remind us that: 

^rogreU^;SiEPr!.rs br i H " ' Assessment of Educational 

conS?r"ntror th^ f responsive to the political 

to take dij acclunt nr;K- ' • V^^ °'^l9inal NAEP design attempted 
that were liLw?* • ^^^^f^'^tins political and social realites 
tnat were 1 kely to jeopardize its successful implementation 

But times have changed and so ha.,e the contexts of educational 
decs ion making. The fear of a national testing program, such as 
NAEP, has largely dissapeared; indeed, M.,EP h&s been critic. red for. 
not delivering more useful Information to educational decisionmakers 
<Messick, Beaton, and Lord, 1983). One of N^.EP's more vocal critics 
In the 1960's, the Council of Chief State School Officers <CCSSO) , 
currently is taking steps to cr-eate its own naMonal testing program 
and to include in its program the capability of providing achievement 
information that will allow comparisdtis amonr; the several states 
<Se,den, 1986). m the 1960's, the organ I z.t 1 on-^and its membership- 
would have considered tl>is anathema. 

The lessening Of fears about a national test i ng program and about 
undue Infringements on loc^l control of the schools has been 
accompanied by other major cf.anges h. the contexts in which education 
decisions are made. Chief among th.se heve been changes In bofch the 
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locus of educational decision making and the cast of educational 
decision makers. We have witnessed an opening up of the decision 
making process and an almost total erasing of public education's 
traditional identity as a separable and special governmental 
operation. These changes have resulted in pulling educational issues 
into the political mainstream; in opening up the system to parents, to 
the general public, to general government, and to special interests; 
and in forcing professional educators to integrate diverse segments of 
the community into the decision making and policy making processes of 
. education. Educational decision making has t^ecome an evalving, 
interactive process open to external Ideas and influences, involving 
many Individuals and groups, involving all levels of government, and 
all levels of organization and program administration. But it is not 
only the locus of educational decision making that has changed. The 
cast of educational decisionmakers has been greatly enl arged-parc., ts , 
othe- citizens, representatives uf interest groups, educators, 
executive staff, legislative staff, legislators, governors, 
Congressional staff all have become participants in the process. 

Central to these major changes in the contexts of educational 
ci.-cision making is an increasino demand for data--data on how students 
are doing In our schools, on what they are learning, on their levels 
of achievement; data on how teachers are doing, on what constitutes • 
good. teaching, on the mix of conditions necessary to ensure that our 
professional teaching ranks become filled with "the best and the 
brightest;- data on curricular programs, on effective instructional 
practices, on new ways of learning; data on the context in which 
schools operate, on the climate in the classroom, on family, <.oclal, 



and economic environments; data on resources, on the most e-ffective 
mixes and uses of resources, on resources and ec»uity issues; data on 
alternative approaches to schooling, on the private sector of 
schooling, *m choice within the public sector. Yet, as we noted 
above, existing national data projects fall, far short of meeting these 
expanding needs for more comprehensive, integrated, and representative 
information. 

In sum, American education is changing rapidly, with new 
educational policies that affect all part i c i pants and stakeholders. 
To understand the need for policy change, its short-term impact, and 
its long-term effects, will -require a radically improved, and vastly 
changed, national education data system. 

Fortunately, ir our view, the upcoming reconfiguration of NAEP 
presents an opportunity "to seize the day," to capitalize on a program 
which already is in place, already moving toward expanding its 
purview, and to use it as the base for building a national data system 
for elementary and secondary education in the United States. Three 
years ago, in presenting the conceptual framework and major features 
of its new design for NAEP, Educat i onal Test i ng Service (ETS) argued 
that its program would have to meet this expanded ne«d for 
information: 

The central question now before the directors of NAEP is how to 
conduct a national assessment that will be di rec t 1 y rel evan t 
to state and lo cal pbl i cy-makers [emphasis added] as well as 
serve as a creditable national indicator of educational 
competence for the general public (Messick, Beaton, and Lord, 
1983, p. 7>. 

Three years prior, Henry Acland (1980) set forth what he saw as the 
major functions of NAEP: <1> to provide an information base for 
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federal policymakers, <2> to establish a data base for research, <3> 
to keep track of performance levels, and <4> to help state and local 
education agencies. This set of objectives, the 1983 ETS observation 
that NAEP must be directly relevant to policy makers at state and 
local levels, and the fact that NAEP already has moved well bc^yohd 
exclusive collection of outcome data, would seem to corroborate the 
argument that NAEP, in the future, can and should become the central 
vehicle for establ I sh i ng a new national data system for elementary, and 
secondary education. 

Under such a configuration, NAEP could provide comprehensive 

information on education-information not only on the outcomes of 

schooling, but also on school setting and the schooling process 
itself. Under such a configuration, NAEP could provide integrated 
information, that is, information that provides linkages between and 
among outcomes, settings, and processes. Under such a configuration, 
NAEP could provide information representative not only of the Nation 
as a whole and its major regions, but also information representative 
of individual states. Under such a configuration, NAEP could provide 
accurate, comparable, and timely information. Under such a 
configuration, NAEP could begin to meet the now largely unmet ne.ds of 
the increasingly broad array of educational policy actors identified 
above . 

An added benefit of moving in this direction would be a • 
definitive resolution of the future role of NAEP and its place among 
the discrete and sometimes competing education data projects mounted 
at the national level. For example, a redesigh of NAEP along the 
dimensions outlined above might negate the need for the separate and 
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costly effort being undertaken by the CCSSO to acquire state-by-sta.te 
information on the outcomes oi schooling. The redesign of NAEP as 
ihe national data system on elementary and secondary education 
also offers promise of resolving the current dilemna of the Center for 
St..tistlcs, namely, whether to continue to tinker with its present 
inadequate "non-system" or to move forward to build the new and 
substantively different system suggested as part of its redesign 
project <Hall, Jaeger, Kearney, and Wiley, 1985). Finally, a redesign 
of NAEP. so that It becomes, In fact, the national data system, .for 
elementary and secondary education holds promise of meeting the 
concerns reportedly voiced in the soor> to be released report of the 
National Academy of Sciences, In which the Academy notes that the 
Center for Statistics currently does not possess the human resources 
needed to mount and sustain a quality national data system. Reliance 
on the private sector rather than a governmental agency for the design 
and operation of the system, as has been the practice with NAEP since 
its Inception, would seem to hold greater promise for the development 
and Implementation of a national data system of the highest quality. 

The Broad Outlines o f a Rernnf j Qured nafp 
li one accepts the argument that serious consideration should be 
Qi.en to building NAEP into the national data system on elementary and 
secondary education, the data system that will meet the expanding 
infomation needs of r broad array of actors Involved in education 
decision making, one. then has to ask, "How might we go about that?" 
In the pages that follow, we attempt to answer this question by laying 
out the broa<i outlines o* what a reconfigured NAEP might look like. 
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First, we.describe a comprehensive conceptual model o-f the schooling 
process; only I f we have a conceptual framework to guide the 
assessment of educational quality can we make appropriate choices 
about what Information to collect and in what form it should be made 
available. Second, we establish a cet of basic criteria which should 
guide the reconfiguration of NAEP If lt is to become the national 
data system on elementar/ and secondary education. Third, we suggest, 
in broad outline, what paths the development and phasing of the NAEP 
redesign might follow. Fourth, we identify some examples of the 
benefits that will be provided by the redesign of NAEP along the" lines 
we propose. Finally, we suggest some next steps that might be taken 
serious consideration Is given to our proposal. 

One major caveat is In order. We do not assume that the 
reconfiguration we are proposing can or should be accomplished 
overnight. Indeed, one of the attractions of developing NAEP into the 
national data system is that NAEP already exists; it represents a 
sWg base on which to build. It can continue to deliver important 
and valuable information even as it systematically moves toward full 
implementation as the national data system. Concurrently, as a 
reconfigured NAEP implements new modules, other existing national data 
projects can be phased out without major upsets in important data 
streams. While we are acutely conscious of the immediate unmet needs 
of data users, we also recognize that we are discussing the design 
of a system that must serve us well mto the 21st Century. 
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A Conceptual Model of the Schooling Process 
NAEP, if it is to serve in the future as the national educational 
data system, must be designed to fulfill the information needs of 
those who have authority and responsibility f or mai nta i n i ng and 
enhancing the quality of the educative process as it takes place in 
the schools and school systems across the land. This group incudes, 
at a minimum, educators, citizens, public officials, and parents. If 
a redesigned NAEP is to meet fully their information needs, then it 
must supply information not only on the outcomes of the educative 
process, but also on the process itself as well as on the contexts in 
Which the process takes place. 

If we are to apprehend fully'this educative process and the 
influences that bear upon It, we must rely upon a conceptual model. 
This model may be simple or complex and i t may be implict or 3xplicit, 
but its existence is a prerequisite to any understanding of the 
effectiveness. and quality of schooling. Our conceptual 
mode l-out Lined below-for describing an educational system focuses 
on the school because it is at the level of the school that 
educational activities take place and that pupils participate in them. 
Our model is drawn from the work of Harn i schf eger and Wiley <1985). 

Fundamentally, schools and communities they serve differ in 
several important ways: 

^' F^i^'l^^ and Community Environmpot . The f ami 1 i es and 
communities served by different schools differ in significant 
ways. They differ in the resources available in the homes of the 
pupils for support of the i r school i ng. They differ in types and 
levels of aspirations parents have for their children. The 
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familx composition of the community ^U^cis the attitudes, 
values, and goals oi a pupil's peers. All of these form the 
context within which schools can educate their pupils. 

2. Educative Dlffirulty . Schools are faced with differences 

In the types and levels of educative difficulties with which 

their pupils present them. Some present handicaps or limited 

proficiencies in English. Others come with limited levels of 

prior learning. Still. others enroll with ccgiMtive 

acconiplishments and capabilities, motivations, ^nd out-of-school 

environments and resources which make the educative efforts 

in some schools easier and less complex than those in other 
schools. 

3. Resources . Schools have available to them different 
levels of monetary resources and different amounts and Kinds of 
non-monetary resources, such as vounteer time, and donated 
supplies and equipment. These resources are exchanged, 
allocated, and configured to provide a teaching staff, 
facilities, educational materials, and the like. 

4. Goals . Schools aspire to distinctive goals. For 
example, some public secondary schools design their entire 
curriculum around . ost-secondary career paths which primarily 
begin in selective colleges and universities, while other 
schools— for example "vocational- ones-may focus their entire 
program around immediate job entry to skilled and semi-skilled 
occupations. 

12 
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5. Process. Schools o-f^er educative experiences -for which 
they. require or encourage pupils' participation. In addition to 
In-class experiences, these in iude work experience, homework. 
Schools also structure these experiences around different 
standards. These standards influence the pursuit of goals with, 
different expectations for performance, differing time allowances 
for accomplishment, and differing criteria for selection into 
subsequent experiences. Schools also differ in the types and 
amounts of participation of their pupils in these educative 
experiences as well as in. the range of experiences made 
available. These variations include differences in selection, 
participation, and compleiion of educational programs, course 
work, and homework as well as differential school attendance. 

6. Outcomes . All through the schooling process, to the 
conclusion of secondary schooling and beyond, schools differ 
greatly in the goal -rel evant accomplishments and achievements of 
their pupils. The^e include cognitive capabilities, credentials, 
and career and life paths generally. 

None of the above areas can be neglected if we are to under-^tand 
schooling in ways that carry meaning for those who participate in it 
and those who are concerned abou t I t and its consequences. School 
outcomes may differ by intent as well as efficacy of programs and 
activities. Schools, school districts, and entire school systems are 
presented with considerable variations in the levels of preparation, 
handicaps, and other educative difficulties that their pupils bring to 
.the schooling process and these have profound consequences for 
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.outccxnes. And schools and the Urger systems within which they are 
embedded really do differ in their effectiveness. Thus it is vital to 
describe, against a conceptual frame-^ork, each of these differences in 
cohesive fashion, as well as to attempt to sort out the reasons for 
differential outcomes against the structure of their origins. 

Figure 1, on the next page, displays such a conceptual framework. 
It focuses on the schooling process, distinguishing teaching 
activities from pupils' exposure to and participation In the resulting 
educative activities. And it traces these aspects of the process to 
their origins— prior and contemporaneous characteristics of pupils, 
community and family expectations, c.rricular goals, and resources-as 
well as linking them to their consequences. Within the context of 
school quality assessment and its bearing on school improvement, this 
conceptual framework treats floals, educative Hi ■ r., i f i , ^^nd 
nesfiurces as pre-conditions or background elements for process 
description and outcome interpretations. 

The Basic Design Criteria 
What, then, are the implications of this conceptual framework for 
the redesign of NAEP as the national data system for elementary and 
secondary schooling? First, in order to meet the information needs of 
the broad array of local, state, and national educational decsion 
makers identified earlier, the data base must be structured to provide 
information on all aspects of the schooling process as described in 
our conceptual model. This means that the data base must be 
S-Sniar^iieD^; put simply. It must be adequate in scope and 
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Figure 1, A Conceptual Frame for the Schooling Procei 
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coverage; it must contain accurate, appropriate, and timely 
Information on (1) the school setting, (2) the schooling process 
itself, and (3) the outcomes of schooling. 

A second major requirement of the data base, in addition to 
being comprehensive. Is that it be integrated , that is, that 
its data sets and elements be linked to one another. The user 
must be able to ask and have answered questions about the 
relationships among background characteristics, the schooling 
process itself, and the outcomes of the process. The data base 
must be able to provide information to answer to such questions- 
as, -What dollars buy what services for which students with what 
results?" Or, "What programs staffed by what types of teachers 
are effective for pupils with particular educative difficulties?" 
Only if the data base is so structured' as to allow relevant 
1 Inkages among its components wi 1 1 the requirement for an 
Integrated educational information system be met. 

Third, the dual requirements for a comprehensive and an 
Integrated system demand, in turn, that data be collected in 
micro-record form , as opposed to macro-record or aggregated 
form. We define a micro-record as a datum on an Individual 
person or entity rather than a datum on a collection or aggregate 
of individual persons or entities. A mi cro-record can be dealt 
with as an individual datum or aggregated; for example, 
individual micro-records on pupils can be aggregated to the 
school level. A macro-record, on the other hand, cannot be 
diaggregated. More Importantly, the micro-record permits of 
linkages with other micro-records; for example, micro-records on 



individual pupils can be linked vgith micro-records on Individual 
teachers and, In turn, with micro-records on speci*ic curricular 
of-ferings In which the teachers and pupils are participating. 
The micro-record format, through Its linkage capability, permits 
the information user to ask questions about relationships among 
the sets that make up the data base. 

The three previous requirements identify basic criteria that 
must be met in establishing a redesigned NAEP data base and for 
getting information Into that data base. A third Issue remains, 
.namelxi what processes will be used for getting information out 
of the data base? Thus, a fourth requirement Is that a national 
educational Information system must be able to deliver 
information of a comprehensive and integrated nature on the 
schooling process In the Nation as a whole, that is it must be 
capable of delivering Information that is nationally 
representative . It must be able to report on the status and 
progress of elementary and secondary schooling In the United 
States. It also must be able to deliver information on 
sub-national or regional populations. NAEP, of course, meets 
this part of the requirment. .But, In addition, we take as a 
given that a national data system for the 1980's, 1990's, and the 
21st Century also must be c..pable of producing information that 
can be used to compare the condition and progress of education In 
the various states; in short, the redesigned system must be 
capable of delivering information that is representat I up n-f 
each o4 the fifty states . 

While this requirement dictates attention to how Information 
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gets into the data. base, e.g., the sampling designs which will be 
employed, i t al so di ct.tes-al ong ui th the previously identified 
requirements o^ comprehensiveness, integration, and micro-record 
formats-n.hat types oi reports must be available to users oi the 
system. Users, with the possible exception of researchers, 
generally will not be interested in micro-records per se but 
rather reports developed from the processing of mi cro-records 
— e.Q., tabulation, aggregation, and analyses. Thus, while 
micro-records represent the form In which Information flows into 
the data base, reports based on processing of the micro-records 
represent the form in which information flows out of the data 
base. Yet, a simple proliferation of reports will not meet the 
needs of the broad array of local, state, and national dec i son 
makers which we i den t i f i ed_ I n the opening pages of this paper. 
A national educational information system must be capable of 
carefully tailoring its reporting formats and mechanisms If it Is 
to serve the particular needs of this broad arr*.y of decision 
makers. Cer tai n dec 1 si on makers, for example Governors, have 
needs for only certain kinds of information and not for other 
kinds; the system must be capable of meeting these needs. In 
short, the system must be capable of screenino .nH ^ 
its reporting form.t>- ...ith the n..H. p. ^ticular 
addition to questions of content, the screen I ng and matching 
require attention to establ i sh i ng the mechanisms necessary to 
actually get the reports to decision makers and decision makers 
to the reports and. In the case of researchers, to the relevant 
portions of the data base Itself. 
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Finally, the development o-f proctsses icr getting information 
out o-f the system has to pay serious attention to timin g. 
Unless the information is ava i 1 abl e when needed, the content and 
form of the reporting mechanism makes little difference. Timing 
Involves setting priorities for reporting different sets of 
information to different users, as well as priorities for 
providing different users access to different sets of 
information. In sum, NAEP, as a national educational information 
system, n.ust be capable of de 1 I ver I ng per i odi c and di f f erent i at.-d 
reports on the status and progress of schooling to a broad array 
of local, state, and national decision makers, as well as making 
available to different users, including researchers, special 
reports on and public use sample^^ relevant to particular aspects 
of elementary and secondary schooling in the United States and in 
the several states. 

These then represent the basic requirements that a redesign 
of NAEP must meet if it is to fulfill its promise of becoming 
the national educational data system. We now reiterate these 
requirements, as well as certain additional requirements, in more 
succinct form and identify them as the basic criteria that we 
believe should guide the future design and development of NAEP. 
The basic criteria are: * 

1. COMPREHENSIVENESS-the system must have a data base 
capable of providing information on all pertient 
aspects of elementary and secondary schooling including 
background characteristics, the school I ng process 
Itself, and the outcomes of schooling. 
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MICRO-RECORD FORMAT-=.n dat» must be collecUd »„d 

de".ed'2.""H°r"°'~'' - micro-record be in. 

fndlv!du:? en?!**""? °" " in-ividu.l person or .n 

rfprfs^tuv^fr *° "''"9 nation.,,. 

..:ur!e^^^•rc^po°:??ro^:^^^^^^^:-:-- . 

ACCURACY-a,, data must be veri + iab,y accurate- thev 

Tnc udtnraidrtf 11,:^'°"^ c^n^HVroc'ld'ures 
rr^i i«-I- f^e interviews as a routine part of data 

cnli!^f'°"' "[1.*'^°^^ °" ^'^t'^y ^f^d data procelsino 

rou%" :e^":r,,^\^"'"P^^*---^dits, and regular and 

routine calculation of measures of variance. 

COMPARABlUTY--data from different Jur i sdi ct i ons must 

of ieoortlnn'tnH definitions; comiJSn un ts 

ot reporting and common definitions are necessary 
precursors of useful data aggregat i oni! 

wit'fh^rff^r"' ?f"^:^^ data must be limited to that 
^Kree mnnfh' "V^*"^' ^"d analyzed within 

three months and reported to policy makers within the 

PRIVACY AND SECURITY^-because some of the files contain 
nformat.on about individuals, e.g., persona 

confident- 'i^r'^T^ longitudinal studtes, strict 

confidentiality and security measures must be in force. 

a'va°Mfbfffr ^^LYSIS-a specif ic schema must be ' - 

repor^tino fnn.^ ? °^ par t i cul ar . users; a wide array of 
avail aM? 1°'^'"^*^ ^«ess mechanisms must be 
availab e to serve the different users: *sDec i fi*- 

tii^Vi :ri:r* "".'.^'ij^' -eung'^^e'dr?::;: . 

timelines imposed by the needs of different users. 

cofls^shl^'^i^^''^^'^''^'^'--* P***-^ °^ shared user 
exclusrveiv nn rn'*',"'" ^^^^^'^J ^^^^er than rely 

!^formlt!ln f support for transmitting 

Information to users and/or providing them access to • 
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.n^orm^t.on, a national educational data system should 
also draw support -from a program oi user fees and 

n.^d! o/^r^^"^' '^^ capacity to to serve the differing 
needs of its users; equally Important, 

transmission/access modes should incorporate the latest 
developments in electronic communications technology! 

Development and Phasing 

Transcendent Develop ment and Phasing I......p<. 

The redesign of NAEh as the new national data system on 
elementary and secondary education would substantially affect current 
data collection, analyis and reporting activities in state education 
agencies, in local education agencies and in the U.S. Department of. 
Education. A new national data system will require dollar resources 
currently allocated to ongoing federal projects and will impose 
respondent burdens that will preclude the continuation of numerous 
existing federal data-collection projects and the intiation of others 
Issues such as these transcend the selection of a data system design, 
and influence the phasing and timing of data system development and 
installation. The most pertinent of these issues are: 
<1) preservation of essential time data series; 

.<2) requirements for research needed to develop critical * 
elements o-f the new data system: tnxicai 

iy^feJlsr.n'd'""* local eduction ageo^y data 

<5.) cost and personnel requ i rements.* 

Preservation of es^pnfi.i f;^. ai though we began 

this paper with an identification of the inadequacies of present 
.atlonal data projects, nev^er the! ess, the Center f or Stat i st I cs over 
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the years has maintained several essential data time series. 
Examples can be found in the Center's publications, The Condition 
of Education and the Digest of Educational Statistics , and 
Include, by way of illustration, total enrollments at all levels of 
education throughout the United States which have been reported since 
1899-1900 <Dme_st of Education;> l Statistics. 1983-84 , Table 3, p. 
8). Such time series must be preserved in the redesign of NAEP into, 
the new national data system. 

Requirmements fo r Research Needed to Develop Critical 
Elements of the New Data System . Most elements of the new 
national data system we are proposing can be developed using existing 
survey and measurement technology. However, other elements 
undoubtedly will test the current state of the survey and measurement 
art and will require intensive research and development. For example, 
the system we. are proposing requires micro-record information on a 
variety of educational outcomes including, but not limited to, 
achievement test data. To secure such data while adequately 
controlling the respondent burden imposed on individual students will 
require the development of new, highly efficient outcome measures and 
new approaches to the use of matrix sampling. As a second example, 
our proposed system calls for timely production of policy-relevant 
analyses that are responsive to immediate and particular requests from 
the broad array of information users including policy makers in all 
levels and branches of government, as well ai the new constituencies 
identifed in the Initial section of this paper. Considerable research 
is needed to develop mechanisms that will enable the NAEP contractor 
to meet these immediate and particular requests for Information with 
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irr«ely, valid, and responsive policy-relevant analyses. Research will 
•e needed. in such areas as verification of the requestor's rights to 
ata access, mechanisms for rapidly and accurately building relational 
nalyses using data stored on the basis of distinct file structures, 
ethods for provididng users with . variety of alternative relational 
nalyses, and methods for assessing the relative utility to policy 
akers of such alternative analyses. 

Provision of AdPonate Time fnn p^ta Sv^t^m Testino and 
grif ication . Because many features of the data system we are- 
roposing are novel, extensive testing and verification of mechanisms 
3r data collection, data aggregation and storage, and information 
^trieval will be required. Such test ing wi 1 1 require a continuing 
Dmmitment of resources by the federal government, a continuing 
Dmmitment by the NAEP contractor, and the continuing cooperation of 
:ate and local agencies over a period of years. Every effort should 
^ made to ensure that no element of the data system will be used 
.erationally until it has been throughly tested and its quality has 
fen verified.- 

Impact on Current State and Lor.l FH.... tion D.t. <,y ^,.n.. 

a state chose to keep i ts own data systems completel'J^ separate from 
:e national data system, no additional data burden would be imposed 
r would the state be required to adapt i ts data-element definitions 

be coincident with those of the national system. On the other 
nd, if a state chose to fully Integrate Its data systems with the 
tional system, It would have to accept and adopt the data-element 
flnltions used In the national system, and I t would have to adopt 
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the micro-record structure that is central to our proposed redesign. 

The .impact of the proposed national data system on local 
education agencies generally would be limited to those local schools 
.nd districts that would be included in the state representative 
samples used by the system in a given year. In sampled schools and 
districts, the volume and density of data collection envisioned for 
:he new national system repesents substantial data burden and likely 
'ould require local agencies to adapt their owu data systems in, 
everal ways. For example, local agencies might choose to make the 
lata element definitions they use consistent with the newly designed 
ational system. Local agencies might also choose to make the 
tructure of their data systems conlstent with the micro-record 
tructure of the national system we are proposing. 

Cost and Personnel Regu i rofnon t^: . m any truly national data 
xstem on elementary and secondary education, a large portion of the 
osts of developing and mai ntai nl ng the system would be borne by the 
.deral government-i n our proposal by contracti ng wl th Independent 
rganizations and agencies selected to design, develop, and Implement 

system. In particular, the costs of necessary research and 
.velopment would be borne entirely by the federal government, as 
JUld the costs of testing and verification. 

State costs and personnel requirements would depend almost 
•tirely on a state's chosen level of participation in the system. To 
.e extent that a state phose to Integrate its data systems with the 
vily designed national system. It would share to some extent in the 
'St of developing the system. Dollar outlay costs and personnel 
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costs Incurred by 1 ocal. educat i on agencies would depend on the degree 
to which they chose to integrate their own data systems with the 
national system— and the degree to which their state chose to 
Integrate its data system with the national system. 

. Spec! -fie Developm ent and Phasind T«^«^iip«^ 

Phases o-f Deuel opment. We propose that the redesign be 
carried out In distinct calendar phases encompassing a five-year 
period. Within each phase, the specific categories and subcategories 
of data elements which would compose the new national data system 

- would be ai different levels of development. Data elements would 
differ In terms of their avaiiability for operational use and the 
level of their aggregation. Some data elements would be objects of 
research and development; other data elements would have advanced to a 
field testing and verification stage; still other data elements would 
have been tested and verified In earlier phases, and would be 
available for operational use; a fourth category of data elements 
would not >et be available in any form. In addition, some data 
elements would be available 1 n 1 1 1 al 1 y on 1 y f rom aggregate records, 
while others would be available in the form of micro-records. The 
phases would also be di st i nqu 1 shed by the numbers and types of data 
elements that could be linked across data categories and data files, 
as micro-records become available for operational use. 

Categories of Data . The categories and subcategories of data 
that would compose the proposed national data system would be drawn 
directly from the conceptual mode T descr 1 bed earlier. The major 
categories Include; environment (community and family 
characteristics and expectations); Incomlno re«m.rcp«. (financial 
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revenues and other incoanng resources for schooling); e ducative " 
difficulties (pupils- capabilities, motivations, handicaps, 
English language facility, out-of-school supports, etc.); 
educative Qoal«> (school goals and objectives, curriculum); 
allocated resources (facilities, staff, equipment, materials and 
other allocated and purchased resources); education.! .u.^u, 
<curricular offerings, standards, teaching- and school -re 1 ated 
*<=tivities), participation , (pupil participation in the process of 
schooling); and outcomes (achievement, graduation or dropping out, 
political part I pat ion; employment). 

. Calendar , i^. we propose that the first phase of 
development begin on July 1, 1987 and extend to December 31, 1987. 
The second phase of development would begin on January 1, 1988 and 
extend to'june 30, 1988. Subsequent phases would encompass six-month 
periods thereafter, through June 30, 1992. 

Status of Data Elements . In any phase, each data element 
that would be a part of the redesigned national data system could be 
characterized as belonging to one of three categories of development. 
At one extreme, would be data elements that are not yet Included in 
any form. An intermediate category would be data elements that were 
collected only In aggregate form, e.g., school membership determined 
from a report prepared by a school . We do not propose that the new ■■ 
system would be responsible for developing such aggregate reports, but 
only for maintaining specific elements that currently are a part of 
critical Center for Statistics data-collection activities until they 
could be replaced by tested and verified mfcro-records. These micro- 
records would constitute the third category-that Is, the other 

Er|c 2S 
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extreme of deve 1 opment-of data elements. Currently, such 
micro-records only exist in data projects such as NAEP and NELS, which 
again argues -for using NAEP as the base for building the new national 
data system. 

Availability of Data Elements . We believe that most, if not 
all, of the data collection formats for elements required in early 
Ehases of implementation of the new system already exist within 
current data programs, and particularly in NAEP and NELS. Development 
of revised data collection formats and activities aimed at 
incorporating existing NAEP and NELS data elements and formats would 
be -required to implement the system, but more fundamental -research" 
activities would not. At any phase of development, however, a data 
element that existed in micro-record form might not yet be available 
for operational use. Initially, some micro-record data elements would 
require extensive research and. development. Any newly developed data 
element would be subject to_ extensive field testing and ver i f i cat i en ; 
that is, any new data element in micro-record form would not become a 
part of the operational data system until convincing evidence of its 
val idity and util i ty had been amassed. 

Linkage Among Data Files . As the different sets of data 
elements become available for operational use, linkages among these 
sets must be establ i shed, tested, and verified. The testing and 
verification also must include assessment of the relative utility of 
the relational policy analyses generated from the linked data sets. 
These assessments must continue and encompass the increasingly larger 
number of linked data sets that come on line as additional 
micro-records become operational, until the new system becomes fully 
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operat i onal . 

Concurrent Development ActiuitiPc . while we will not 
elaborate here, implicit in the research, development, testing and 
v.eriflcation activities outlined above are such data system design and 
development activities as the Identification of essent i al popul at i ons 
of general iazatlon; the design and selection of samples of data 
suppliers; the design of mechanics for the collection of data, 
including specifications for, recruitment of, and training of data 
collectors; the design of survey field procedures; the deslgnof plans 
for the analysts of data and reporting of results; the development of 
systems for transmission of data; and the development of software 
systems for data receipt, control, editing, analysis, and 
summarization. In Table 1, on the following page, we provide a 
truncated outline illustrative of the development and phasing 
activities outlined above. For example, under the category 
"Environment- we identify the sub-category "community and family 
characteristics- and indicate that, at present, data in this 
sub-category are collected only in aggregate form in the data 
collection activities currently being conducted by the Center for 
Statistics. - In Phase I, research and development activities would be 
undertaken; in Phase II, testing and verification wouldtake place; 
and in Phase III, the data would become available in micro-record 
form. The remainder of Table 1 can be read in the same fashion. 
In the sub-category -school goals," for example, research and 
development activities would not begin until Phase III, whereas in the 
sub-category -dropouts'-where data In aggregate form already are 
being col 1 ected— research and development activities would begin 
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labkl. niuslrative Status of Categories nnd Subcaie«;ories of Data at Various- 
Phases of Development of the National Data Svstcm 







Status of Data Catepory 




Caiecories& 
Subcatcffories 


Current 


Phase I 


Phasf U 


Pha^ip TTT 


Environment: 

Family and commun. 
characteristics 


Ag. 


Ag.(R&D) 


Ag. (T.&V) 


Micro. 


Resources: 

financial revenues 


Ag. 


Ag. 


Ag. (R&D) 


Ag.(T&V) 


Educative Difficulties: 
Handicapped status 
Motivation 


Ag. 
Non. 


Ag.(R&D) . 
Nor. T?.&D) 


Ag.(T&V) 
Non.(R&D) 


Micro. 
Non.(T.&V) 


Educative Goals: 
School goals 


Non. 


Non. 


Non' 


Non.(R&D) 


Participation: 

Course enrollment 


Micro. 


Micro.(R&D) 


Micio.(T&V) 


Micro.(Rev.) 


Outcomes: 
Dropouts 


Ag 


Ag.(R&D) 


Ag.(T&V) 


Micro. 



LEGEND: 
Non. 



Ag. 



Micro. 

(R&D) 
(T&V) 
(Rev.) 



denotes a data subcategory that does not presently exist in the set of projects 
operated by the Federal Center for Statistics. 

denotes a data subcategory in which data arc presently coUected only in aggregate 
form in the set of projectis operated by the Federal Center for Statistics. " 

denotes a data subcategory in which data are presenUy coUected in the form of 
microrecords m the set of projects operated by the Federal Center for Statistics. 

denotes a data subcategoiy in which research and development is to lie conducted. 

denotes a data subcategory in wJiich testing aiid vcrficarion is to be conducted. 

denotes a data subcategory in which the data previously existed in the form ♦ 
indicated, but for which revised data elements arc developed and adopted. 
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n Phase I... Note that Table 1 illustrates only the -first 
proposed deMelopment phases. 

Priorities -for Development and Phasing 

Is only Illustrative of the types of decisions that would 
de in the redesign o-f NAEP into the national data system 
y and secondary education. The actual choices o-f the 
h categories and sub-categories o-f data elements would be 
Id need to be made by the U.S. Department of Education 
ncert with representat i ves" of state and local education 
contractors selected tc design and Implement the system, 
ropose consideration of the following priorities, 
eptual model defined earlier in this paper provides 
data that are required to meet the information needs of 
Icy-makers at several levels of government, as well as 
r>ew constituencies -for information identified in the 
of the paper. Of al 1 data categories defined by that 
process information is least available now. School 
nation includes: information on the educative goals of 
on al located resources—fac i 1 i tes, staff, equipment, and 
formation on educational pursuits, curricular offerings, 
iching-related and school -rel ated activities; and 
1 pupil participation In the process of schooling. There 
tical need for high quality outcome data. The best 
ivailable are presently provided by NAEP — again 
of the argument that NAEP is particularly wel 1 -si tuated 
base of a new national data system. However, under 
jements, NAEP data are limited to students at relatively 
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few grade levels, are only collected biennially, and are limited in 
subject matter tested. Thus, in our view, two categories of data- 
school process data and ou tcome data— deserve pri or 1 ty attention in 
the development of the proposed system. 

Yet, although we believe that information on process and outcome 
should have the highest priorities in terms of information needs, we 
also believe the new data system should attempt first to build 
micro-records on a small sub-set of data In order to develop the 
collection process and refine the data base development process. 
Information on pupil participation which would provide data for 
enrollment and attendance would be our priority candidate for Initial 
development. At the same time, the research effort to develop a more 
comprehensive set of process and outcome data should be giOen high 
• priority, as we argued above, and proceed on a parallel track. 

School context information should constitute a third area of 
priority development, particularly information which describes the 
environment in which schools operate, such as community and family 
characteristics and expectations, as well as information that 
describes the eduative difficulties of students. In our judgement, 
these two categories of data should receive attention once the 
development of micro-records is well underway in the school process 
and outcomes categories. 

Our fourth order of priority would be to address data needs in 
the educative goals category. A final priority, but certinly 
essential, would be the categories of incoming and allocated 
resources, i ncl udi ng revenues, and expenditures for, and stocks of, 
materials, equipment, f ac i 1 i t 1 es, and personnel. As is clear from 
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Table 1, above, certai n. ex i st i ng aggregates are recommended -for 
phasing Into micro-record formats in stages beyond Phase I. This 
raises the Issue of parrallel aggregate reporting for existing 
aggregate data series to allow users to move from the old problematic 
series Into the micro-record based series. This overlap will need to 
be carefully planned Into the phasing of the new system. 

Some Benefits of the Redesigned System 
The consumers of Information about educational systems include 
parents concerned about the education of their children, citizens, 
worried about the qual 1 ty- and and efficiency of the education their 
tax dollars finance, professional educators making decisions about 
programs and pupils, and public of f i c i'al s desi r i ng to design laws, 
requirements, and resource al 1 ocat I ons wh I ch wi 1 1 effectively improve 
education. All of the^e consumers are concerned that the information 
which reaches them be relevant and useful. to their needs, timely, and 
accurate. 

Common to all of the consumers are concerns about quality and 
effectiveness. It is- this i nf ormat i on wh i ch is most desired in ti.e 
public debate over education. Parents want to know about the quality 
of educational alternatives available to them. Citizens and public 
off icials wish val id assessments of efficiency to know that resource 
allocations are wisely made and carried through to desired outcomes. 

Resource flows are important Information for public officials in 
making determinations of how much and how to allocate resources. 
Federal officials have special concern for how federal resources are 
channeled to pupils and the Impact of these resources on pupils with 
specific characteristiics. State officials, in fulfilling their 
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responsibn I ties, have been modi-fying state educational systems in 
ways that require comprehensive information about participation in 
programs, courses, and other services, about standards of performance, 
and about actual outcomes. Local officials are newly concerned that 
they are eff ec t i ye 1 y mon i tor i ng service delivery, participation and 
ach 1 evement . 

An effectively integrated system— i ncorporat i ng the micro-data 
and records necessary to produce these new types of i nf ormat i on— i s 
needed by al 1 concerned parties. The benefits of a cohesive system of 
the type we are proposing, capable of producing national and state 
comparable data, would be far reaching. Not only would the nvajor^Ht/ 
of consumers of educational information be provided with relevant, 
integrated-, timely, and accurate information at these two levels, but 
the establishment of such a system would produce similar changes in 
district-level information systems. This, in turn, would increase the 
comprehensiveness and comparabi 1 i ty of the information about education 
taking place in local communities. Thus, in our view, the proposed 
redesign of NA£P into the national educational information system on 
T elementary and secondary education, as it becomes established at 
national and state levels, will introduce cohesion in the total 
system. 

Possible Next Steps 
This present paper, at best, has sketched only the broad 
outlines of a proposal to build NAEP into the national data system 
on elementary and secondry education. Much more specific and detailed 
attention needs to be given to the issues and areas outlined in the 

ERIC"..v,.:;^,^^^^^^^^ ......V.-':: , • ■ C V 
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paper, as well as to a good many more issues and areas not mentioned 
or only lightly touched upon, \i such a proposal is given serious 
consideration in the redesign o-f NAEP. For example, the cooperation 
and collaboration of state and local education agencies in the design 
and development of the system is critical. Mechanisms for effectively 
bringing about such cooperation and collaboration would need to be 
established early on In order to ensure the success of the effort. 

Fortunately, a relatively detailed plan for the development of a 
comprehensive. Integrated, representative, accurate and timely, 
national educational information system already exists <See, Hall, 
Jaeger, Kearney, and Wiley, 1985). This plan was developed as part of 
the NCES Data Redesign Project launched In early 1985, is based in 
part on extensi ve . rev lew of the forty or more invited papers subo^itted 
to the Redesign Project (Silverman and Taueber, 1985), and sets forth 
a general blueprint for a fundamentally new national data system for 
elementary and secondary education. It can provide further guidance 
for those who would give serious consideration to the argument that 
NAEP should form the base and core of such a system--a system capable 
not only of meeting information needs In the 1980's and 1990's, but 
also serving the needs of education decision makers well into the 21st 
Century. 
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